Module 6: Mixing Categorical and Continuous Predictors

Working Example

• Last week, we discussed an example where researchers (Baumann et al.) sought to determine how children’s reading comprehension scores after an intervention (i.e., posttest scores) differed by treatment group (control, DRTA, or TA)

Categorical Predictor Variables

• We saw how we could use dummy coding to evaluate whether type of intervention is a meaningful explanatory variable of posttest scores
  • Where \(D1\) represents the mean difference between DRTA and control groups, and
  • \(D2\) represents the mean difference between TA and control groups
• D1 and D2 then become the predictors in a multiple regression model (instead of the original grouping variable, e.g., group):

\[\hat{Reading \ score}_i = {\color{deeppink} {\beta_0}} + {\color{darkcyan} {\beta_1}}{\color{darkgrey} {D1_i}} + {\color{gold} {\beta_2}}{\color{lightblue} {D2_i}}\]

\[\hat{Reading \ score}_i = {\color{deeppink} {6.68}} + {\color{darkcyan} {3.09}}{\color{darkgrey} {D1_i}} + {\color{gold} {1.09}}{\color{lightblue} {D2_i}}\]

\[\hat{Reading \ score}_i = {\color{deeppink} {6.68}} + {\color{darkcyan} {3.09}}{\color{darkgrey} {(DRTA \ vs. \ Control)_i}} + {\color{gold} {1.09}}{\color{lightblue} {(TA \ vs. \ Control)_i}}\]

Adding More Predictors

• Given that the dummy coding approach discussed last lecture are analogous to a one-way ANOVA, we could think of an MLR model with at least one categorical predictor and at least one continuous predictor as an ANCOVA (Analysis of Covariance)!

• So, an ANCOVA model is just a certain type of multiple regression model that includes both categorical and continuous predictors…

• Typically, ANCOVA is used for comparing group means on an outcome variable while controlling for some continuous variable

• It’s common for researchers to use the word “covariate” when referring to the continuous variable in ANCOVA
  • What they often mean by that is a variable that they cannot manipulate and has little substantive or theoretical interest
  • But, as we learned, added variables in the model can be anything we want to condition on or control for statistically (e.g., confounders, forks, colliders)
  • i.e., you simply want to partial out the effects of the added variable in the analysis

Nice Meeting You, Ann Cova!

• If ANOVA/regression with a categorical variable is commonly presented as a method for comparing group means, ANCOVA is often presented as a method for comparing adjusted means across groups (AKA conditional means)

• The interpretation of the slope associated with the categorical predictor will change slightly, but you’re already familiar with this change!

• This will not be different than the difference between any other predictor in SLR when you move to MLR

One More Thing, Ann

• But, there’s this other thing…

• ANCOVA has an additional critical assumption which is known as homogeneity of regression, or parallelism

• Parallelism means that the relationship between the continuous variable, \(X\), and the outcome, \(Y\), is assumed to be constant, or homogeneous, across the levels of the categorical predictor

• Put another way, the traditional ANCOVA model assumes that there is no interaction between \(X\) and the categorical variable

• But, if we reframe ANCOVA as multiple regression, it’s easy to relax this assumption by including an interactions term between \(X\) and the dummy-coded variables representing group membership (we will address interactions in Module 7)

• Another way to think about the parallelism assumption from a regression framework is the assumption that your model is properly specified

  • i.e., you didn’t “miss” an interaction that exist in the population/data-generating mechanism

Parallelism Assumption

ANCOVA Example

• Yes, the reading comprehension example, again!

• Baumann et al. (1992) were actually interested in how the groups differed in their post-intervention reading test scores over and above any differences on a reading test score administered before the intervention

• This is an example of a classic “Pre-Post” research question for which ANCOVA is often applied:

  • “How do treatment groups differ on posttest scores, controlling pretest scores?” Or,
  • “Would the groups differ on the posttest if they had been equivalent on the pretest?”

• Our regression/ANCOVA model will provide estimates of the adjusted mean differences, controlling for pre-test score

Change Scores

• Another approach with pre-post designs, is to model the difference between post-test and pre-test scores such that the outcome variable is the difference betwe post and pre rather than posttest and control for pretest:

  • \(Y_{change} = Y_{post_i} – Y_{pre_i}\)
  • These are called change scores, difference scores, or gain scores

The advantages of using change scores are:

• Logical – you directly model the amount of change from pre to post test
• Unbiased when there are real group differences at pre-test (e.g., non-randomized experiments or important differences that shouldn’t be ignored)

Limitations of change scores include:

• Slightly less powerful when random assignment is used
• More likely to have floor or ceiling effects, which creates a lot of measurement error (i.e., unreliability)

Change Scores vs. ANCOVA

• Although ANCOVA and change scores represent different approaches to handling the same research design, researchers may actually arrive at different conclusions if they were to use both approaches on the same dataset; this phenomenon is called Lord’s Paradox (Lord, 1967)

• Lord’s original experiment was meant to evaluate how young men and women differ on weight change over the course of a semester

• But, men obviously start at a much higher average weight than women

• In Lord’s dataset, men and women do not change at all over time (mean change = ~0 in both groups). Here’s a simulated illustration…

Lord’s Paradox

Lord’s Paradox

summary(lm(change ~ gender, dat))   # Change Scores (t-test)

Call:
lm(formula = change ~ gender, data = dat)

Residuals:
     Min       1Q   Median       3Q      Max 
-15.5233  -3.3977   0.6239   3.9303  13.8364 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)   1.1692     0.5190   2.253   0.0254 *
genderMen     0.1209     0.7340   0.165   0.8694  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 5.19 on 198 degrees of freedom
Multiple R-squared:  0.0001369, Adjusted R-squared:  -0.004913 
F-statistic: 0.02711 on 1 and 198 DF,  p-value: 0.8694

Lord’s Paradox

summary(lm(final ~ gender + initial, dat)) # ANCOVA 

Call:
lm(formula = final ~ gender + initial, data = dat)

Residuals:
    Min      1Q  Median      3Q     Max 
-9.4746 -1.9902  0.1751  1.8293  8.0446 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 33.79411    1.67270   20.20   <2e-16 ***
genderMen   13.42333    0.79429   16.90   <2e-16 ***
initial      0.44538    0.02797   15.92   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 3.006 on 197 degrees of freedom
Multiple R-squared:  0.9463,    Adjusted R-squared:  0.9457 
F-statistic:  1734 on 2 and 197 DF,  p-value: < 2.2e-16

Lord’s Paradox

• So, which approach is correct?! Both approaches are correct!

The two approaches actually answer different research questions:

• Change scores addresses the question, “What is the difference in weight change between men and women?”

• ANCOVA answers, “Is there still a difference between men and women in their final weights, after accounting for where each individual started (initial weight)?”

• Here’s a wonderful “tutorial” by Michael Clark showing more on Lord’s Paradox